Binary Neural Networks Algorithms, Architectures, and Applications (Baochang Zhang, Sheng Xu, Mingbao Lin etc.)

Introduction

FIGURE 1.1

Modulation process based on an M-Filter.

(e.g., sign function) for the reconstructed weights. While the reconstruction is binarized,

the computation in the latent factorized space is done in the real domain. This has several

advantages. First, the latent factorization enforces a coupling of ﬁlters before binarization,

which signiﬁcantly improves the accuracy of trained models. Second, during training, the

binary weights of each convolutional layer are parametrized using a real-valued matrix or

tensor decomposition, while during inference, reconstructed (binary) weights are used.

Instead of using the same binary method for weights and activations, Huang et al. [93]

believe that the best performance for binarized neural networks can be obtained by applying

diﬀerent quantization methods to weights and activations. They simultaneously binarize the

weights and quantize the activations to reduce bandwidth.

ReActNet [158] proposes a simple channel-wise reshaping and shifting operation for the

activation distribution, which replaces the sign function with ReAct-Sign, and replaces the

PReLU function with ReAct-PReLU. The parameters in ReAct-Sign and ReAct-PReLU

can be updated.

Compared to XNOR-Net [199], both HORQ-Net [138] and ABC-Net [147] use mul-

tiple binary weights and activations. As a result, HORQ-Net and ABC-Net outperform

XNOR-Net on binary tasks, but they also increase complexity, which goes against the ini-

tial intention of BNNs. New neural networks that perform better and retain the advantage

of speediness are waiting to be explored. MCN [236] and LBCNN [109] proposed new ﬁlters

while quantizing parameters and introducing a new loss function to learn these auxiliary

ﬁlters.

1.1.4

Structural Design

The basic structure of networks such as BinaryConnect [48] and BinaryNet [99] is essentially

the same as traditional CNNs, which may not ﬁt the binary process. Some attempts have

been made to modify the structure of BNNs for better accuracy.

FIGURE 1.2

MCNs convolution.